Erroneous detection determination apparatus and erroneous detection determination method

ABSTRACT

An area extraction unit ( 2 ) extracts an eye area in an image of a driver. An eyelid detection unit ( 3 ) detects an eyelid in the eye area. A reliability calculation unit ( 4 ) calculates an eyelid reliability by using luminance information of the eye area and positional information of the eyelid detected by the eyelid detection unit ( 3 ). A determination unit ( 5 ) determines that the eyelid has not been properly detected, when the eyelid reliability is less than a first threshold value.

TECHNICAL FIELD

The present invention relates to an apparatus for determining whether aneyelid has been erroneously detected.

BACKGROUND ART

A driver monitoring apparatus for imaging a vehicle interior to therebydetect failing asleep, sleepiness or the like of a driver, is known.When the driver wears eyeglasses or sunglasses (hereinafter, “sunglass”is also referred to as “eyeglass”), a case may arise where an eyelid isnot properly detected due to reflection of scenery in an eyeglass lenssurface, hiding of the eyelid by an eyeglass frame, or the like. If theeyelid is not properly detected, capability of detecting the fallingasleep, sleepiness or the like of the driver is degraded.

In Patent Literature 1, there is described an arousal degree estimationapparatus that, when an eye(s) is erroneously detected, estimates anarousal degree while taking information of the erroneous detection intoconsideration. The arousal degree estimation apparatus of PatentLiterature 1 detects gray levels of pixels along each pixel row in thevertical direction in an image, to thereby extract a pixel group of theeye, by use of the fact that the gray levels in a region correspondingto the eye locally become lower than the other. In addition, the arousaldegree estimation apparatus of Patent Literature 1 determines whetherthe detected eye(s) is due to erroneous detection. For example, thearousal degree estimation apparatus of Patent Literature 1 determineserroneous detection by using a positional relationship of the detectedright and left eyes, such as a distance between the right, and lefteyes.

CITATION LIST Patent Literature

Patent Literature 1: Japanese Patent Application Laid-open No.2007-34436

SUMMARY OF INVENTION Technical Problem

However, according to the arousal degree estimation apparatus of PatentLiterature 1, accuracy is low in determining whether or not the eye (s)is erroneously detected. For example, even in the case where scenery isreflected in an eyeglass lens surface around the eye and the reflectedscenery is erroneously detected as an eye, the eye will not bedetermined to be erroneously detected if determination of erroneousdetection is performed using a positional relationship of the right andleft eyes as in Patent Literature 1. Thus, it is determined that the eyehas been properly detected, so that processing will be performed whileusing the reflected scenery as an eye. The erroneous detection of an eyeis also, and ultimately, the erroneous detection of eyelids.

This invention has been made to solve the problem as described above,and an object thereof is to provide an erroneous detection determinationapparatus which can improve accuracy in determining whether or not aneyelid has been erroneously detected.

Solution to Problem

An erroneous detection determination apparatus according to theinvention is characterized by including: an image acquisition unit foracquiring image data indicating an image of a driver; an area extractionunit for extracting an eye area in the image by using the image dataacquired by the image acquisition unit; an eyelid detection unit fordetecting an eyelid in the eye area extracted by the area extractionunit; a reliability calculation unit for calculating an eyelidreliability by using luminance information of the eye area extracted bythe area extraction unit and positional information of the eyeliddetected by the eyelid detection unit; and a determination unit fordetermining that the eyelid has not been properly detected, when theeyelid reliability calculated by the reliability calculation unit isless than a first threshold value.

Advantageous Effects of Invention

According to this invention, since the determination is made on thebasis of the eyelid reliability calculated by using the luminanceinformation of the eye area and the positional information of thedetected eyelid, it is possible to improve accuracy in determiningwhether or not the eyelid has been erroneously detected.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a configuration of an erroneous detectiondetermination apparatus according to Embodiment 1.

FIG. 2A is a diagram showing an example of an eyelid line; and FIG. 2Bis a diagram when the eyelid line of FIG. 2A is plotted.

FIG. 3 is an illustration diagram of luminance vectors.

FIG. 4 is a graph showing a relationship between an eyelid evaluationvalue and an eyelid reliability.

FIG. 5A is a diagram showing condition around a driver's eye in a casewhere no scenery is reflected in an eyeglass lens surface; and FIG. 5Bis a diagram showing condition around the driver's eye in a case wherescenery is reflected in the eyeglass lens surface.

FIG. 6A is a diagram showing an example of an eyelid line; and FIG. 6Bis a diagram when the eyelid line of FIG. 6A is plotted.

FIG. 7A and FIG. 7B are diagrams each showing a hardware configurationexample of the erroneous detection determination apparatus according toEmbodiment 1.

FIG. 3 is a flowchart showing an example of processing by the erroneousdetection determination apparatus according to Embodiment 1.

FIG. 9 is a flowchart precisely showing an example of processing in StepST4 in FIG. 8.

FIG. 10 is a flowchart showing a modified example of calculationprocessing of the eyelid reliability.

FIG. 11A is a diagram showing a case where, in an eyeglass lens surface,scenery is reflected in a band shape under an eye; and FIG. 11B is adiagram showing a case where, in the eyeglass lens surface, sceneryhaving a contour similar to an eye is reflected.

FIG. 12 is a diagram showing an example of temporal transition ofscenery reflection and eyelid reliability.

FIG. 13 is a diagram corresponding to a case where scenery reflectionsshown in FIG. 12 have occurred, and showing an example of temporaltransition of a ratio of images each with the eyelid reliability of lessthan a second threshold value, in previous images.

FIG. 14 is a diagram showing eyelid reliabilities calculated inaccordance with the processing shown in FIG. 10, in the case wherescenery reflections shown in FIG. 12 have occurred.

DESCRIPTION OF EMBODIMENTS

Hereinafter, for illustrating the invention in more detail, anembodiment for carrying out the invention will be described withreference to the accompanying drawings.

Embodiment 1

FIG. 1 is a diagram showing a configuration of an erroneous detectiondetermination apparatus 10 according to Embodiment 1. The erroneousdetection determination apparatus 10 includes an image acquisition unit1, an area extraction unit 2, an eyelid detection unit 3, a reliabilitycalculation unit 4, a determination unit 5 and a state determinationunit 6.

The image acquisition unit 1 acquires image data indicating an image ofa driver. In the following, description will be made on a case, as anexample, where the erroneous detection determination apparatus 10 isinstalled in a vehicle. In this case, from an unshown camera that isinstalled in the vehicle and serves to image the driver, the image dataindicating the image of the driver is outputted.

The image acquisition unit 1 outputs the acquired image data to the areaextraction unit 2.

Using the image data acquired by the image acquisition unit 1, the areaextraction unit 2 extracts an eye area in the image indicated by theimage data. The eye area is an area that includes at least an eye, andis smaller than the face area. An image processing technique forextracting, from an image of a person, an eye area of the person is apublicly known technique, so that the details thereof will be omitted.For example, an image processing technique is used in which the facearea of a person is specified and a horizontally-long area defined in anupper region in the face area is determined as an eye area of theperson.

Out of the image data acquired by the image acquisition unit 1, the areaextraction unit 2 outputs partial image data corresponding to the eyearea of the driver, to the eyelid detection unit 3 and the reliabilitycalculation unit 4.

Using the image data outputted from the area extraction unit 2, theeyelid detection unit 3 detects an eyelid in the eye area extracted bythe area extraction unit 2. An image processing technique for detectingthe eyelid is a publicly known technique, so that the details thereofwill be omitted. For example, such an image processing technique ascited in Patent Literature 1 is used in which gray levels of pixels aredetected along each pixel row in the vertical direction in an image, andpositions at which the gray levels change locally are detected ascorresponding to the eyelid.

The eyelid detection unit 3 outputs positional information of thedetected eyelid to the reliability calculation unit 4 and the statedetermination unit 6. The positional information of the eyelid isinformation by which an eyelid line as shown, for example, in FIG. 2A,FIG. 2B, FIG. 6A or FIG. 6B to be described later, can be determined.For example, the positional information of the eyelid contains: acoordinate position of the inner corner of the eye, in the eye area; acoordinate position of the outer corner of the eye, in the eye area; anda mathematical function representing the shape of an eyelid line.

Using luminance information of the eye area extracted by the areaextraction unit 2 and the positional information of the eyelid detectedby the eyelid detection unit 3, the reliability calculation unit 4calculates an eyelid reliability. The luminance information of the eyearea can be determined using the image data outputted from the areaextraction unit 2, by the reliability calculation unit 4. The eyelidreliability is an index indicating reliability as to whether the eyeliddetected by the eyelid detection unit 3 is a real eyelid. This meansthat the higher the eyelid reliability, the more correctly the eyeliddetection unit 3 has detected the eyelid. The reliability calculationunit 4 outputs the calculated eyelid reliability to the determinationunit 5.

FIG. 2A, FIG. 2B, FIG. 3 and FIG. 4 are diagrams for explaining anexample of a calculation method of the eyelid reliability.

Using the positional information of the eyelid outputted from the eyeliddetection unit 3, the reliability calculation unit 4 determines, forexample, an eyelid line L1 as shown in FIG. 2A. The eyelid line L1indicates the eyelid detected by the eyelid detection unit 3. One end ofthe eyelid line L1 corresponds to the inner corner of the eye and theother end thereof corresponds to the outer corner of the eye.

The reliability calculation unit 4 plots the eyelid line L1 on the eyearea. At this time, by referring to, for example, the coordinatepositions of the inner and outer corners of the eye in the eye area, thecoordinate positions being contained in the positional information ofthe eyelid, the reliability calculation unit 4 plots the eyelid line L1on the eye area. The eye area corresponds to an image indicated by theimage data that the reliability calculation unit 4 has acquired from thearea extraction unit 2.

FIG. 2B is a diagram when the eyelid line L1 is plotted on the eye area.FIG. 2A and FIG. 2B show an eyelid line and an eyelid-line plotted statein the case where condition around the driver's eye is as shown in FIG.5A to be described later, in which an eyeglass frame F is also shown. Asshown in FIG. 2B, the eyelid line L1 is well-matched to the actualeyelid. The reliability calculation unit 4 calculates a normal vector Vawith respect to the eyelid line L1, for each of the pixels on the eyelidline L1. The normal vectors Va with respect to the eyelid line L1 are asshown in FIG. 2A.

In addition, the reliability calculation unit 4 calculates a luminancevector Vb, for each of the pixels on the eyelid line L1. The reliabilitycalculation unit 4 determines the luminance information of the eye areaby using the image data outputted from the area extraction unit 2, anduses the information for the calculation of the luminance vectors Vb.FIG. 3 is an illustration diagram of the luminance vectors Vb. Theluminance vector Vb is a vector indicating an increasing direction ofluminance. For example, the luminance vector Vb of a pixel P1 isprovided as a vector directed therefrom to a pixel P2 that is the onlywhite pixel among pixels surrounding the pixel P1. Mote that a luminancevector Vb of a pixel that is surrounded using multiple white pixels isprovided as, for example, a vector that passes through a barycenter ofthe multiple white pixels. For example, the luminance vector Vb of apixel P3 that is surrounded using two white pixels P2, P4, passesbetween the pixel P2 and the pixel P4 as indicated in FIG. 3 by a brokenline. Note that the normal vector Va and the luminance vector Vb eachhave, for example, a length of a unit vector.

The more the eyelid detected by the eyelid detection unit 3 is matchedto the actual eyelid, the higher the degree of similarity between thenormal vector Va and the luminance vector Vb becomes. Thus, thereliability calculation unit 4 calculates an eyelid evaluation valueindicating the degree of similarity between the normal vectors Va andthe luminance vectors Vb. It is meant that the larger the eyelidevaluation value, the more similar the normal vectors Va and theluminance vectors Vb are to each other. The eyelid evaluation valuecorresponds to, for example, inner products of the normal vectors Va andthe luminance vectors Vb. The reliability calculation unit 4 calculatesthe inner product of the normal vector Va and the luminance vector Vb,for each of the pixels on the eyelid line L1, and determines a totalvalue of the inner products as the eyelid evaluation value. Note thatthe eyelid evaluation value may be an evaluation value based onsomething other than the inner product, so long as the eyelid evaluationvalue indicates a degree of similarity between the normal vectors Va andthe luminance vectors Vb. For example, the eyelid evaluation value maybe an inverse number of a total value of angles formed by the normalvectors Va and the luminance vectors Vb.

After calculation of the eyelid evaluation value, the reliabilitycalculation unit 4 converts the calculated eyelid evaluation value tothe eyelid reliability. FIG. 4 is a graph showing a relationship betweenthe eyelid evaluation value and the eyelid reliability. When the eyelidevaluation value is not less than a fourth threshold value, the eyelidreliability is calculated according to the eyelid evaluation value. Incontrast, when the eyelid evaluation value is less than the fourththreshold value, the eyelid reliability is calculated to be a minimumvalue, for example, zero. Note that, when the eyelid evaluation value isless than the fourth threshold value, the reliability calculation unit 4may decrease, instead of uniformly setting the eyelid reliability to theminimum value, the eyelid reliability in a stepwise manner according tothe eyelid evaluation value. Further, the relationship between theeyelid evaluation value and the eyelid reliability may be a relationshipthat represents direct proportion therebetween, instead of that in FIG.4. Further, the reliability calculation unit 4 may use as the eyelidreliability, the eyelid evaluation value as it is. In short, variousmethods are conceivable for converting the eyelid evaluation value tothe eyelid reliability.

The example shown in FIG. 2A and FIG. 2B corresponds to, for example,the case where condition around the driver's eye is as shown in FIG. 5A.In FIG. 5A, no scenery is reflected in the eyeglass lens surface, sothat the eyelids appear clearly in the image.

In contrast, FIG. 5B shows condition around the driver's eye, thecondition being different to that in FIG. 5A. In FIG. 5B, although aneye-opening state of the driver is similar to that in FIG. 5A, sceneryis reflected entirely in the eyeglass lens surface, so that the eyelidsappear in a faded state in the image. In addition, for example, an edgeA of a dashboard is reflected in the eyeglass lens surface. With respectto the condition shown in FIG. 5B, the eyelid detected by the eyeliddetection unit 3 is provided as, for example, an eyelid line L2 shown inFIG. 6A. FIG. 6B is a diagram when the eyelid line L2 is plotted on theeye area. The eyelid line L2 is plotted at a position apart from theactual eyelids. Namely, the eyelid detection unit 3 has erroneouslydetected the edge A of the dashboard as the eyelid. In this case, ateach of the pixels on the eyelid line L2, the degree of similaritybetween the normal vector Va and the luminance vector Vb is low, so thatthe eyelid evaluation value, and thus the eyelid reliability, is loweras compared with the case of FIG. 5A.

Using the eyelid reliability outputted from the reliability calculationunit 4, the determination unit 5 determines whether the eyelid detectionunit 3 has properly detected the eyelid. Specifically, when the eyelidreliability calculated by the reliability calculation unit 4 is lessthan a first threshold value, the determination unit 5 determines thatthe eyelid has not been properly detected. The first threshold value ispreset to a value higher than the eyelid reliability calculated in suchcondition as shown in FIG. 5B, for example.

The determination unit 5 outputs its determination result to the statedetermination unit 6.

Using the positional information of the eyelid detected by the eyeliddetection unit 3, the state determination unit 6 calculates an eyeopen-closed state. For example, the state determination, unit 6calculates an eye-opening degree as the eye open-closed state. Theeye-opening degree is calculated by a publicly known method. Forexample, the eye-opening degree is calculated using a flatness ratio ofthe eyelid line determined from the positional information of the eyeliddetected by the eyelid detection unit 3. Further, using thedetermination result outputted from the determination unit 5, the statedetermination unit 6 determines whether to make the calculatedopen-closed state valid or invalid. The open-closed state determined tobe valid is to be treated as that which adequately represents theopen-closed state of an actual eye of the driver. In contrast, theopen-closed state determined to be invalid is to be treated as thatwhich does not adequately represent the open-closed state of the actualeye of the driver. Accordingly, the open-closed state determined to beinvalid is to toe discarded immediately, for example.

It is noted that the state determination unit 6 may be configured to toecapable of calculating an eye direction or an arousal degree. Whenhaving determined that the open-closed state is valid, the statedetermination unit 6 may calculate the eye direction. Further, whenhaving determined that the open-closed state is valid, the statedetermination unit 6 may calculate the arousal degree. The arousaldegree is an index indicating a degree of arousal of the driver, and iscalculated on the basis of a temporal variation of the eye-openingdegree, as exemplified by a case where, when the eye-opening degreeseach corresponding to an eye that is closed are calculated successively,an arousal degree indicating the failing asleep is calculated.

Next, using FIG. 7A and FIG. 7B, hardware configuration examples of theerroneous detection determination apparatus 10 will toe described.

Functions of the image acquisition unit 1, the area extraction unit 2,the eyelid detection unit 3, the reliability calculation unit 4, thedetermination unit 5 and the state determination unit 6 of the erroneousdetection determination apparatus 10 are implemented by a processingcircuit. The processing circuit may be dedicated hardware, or may be aCentral Processing Unit (CPU) which executes a program stored in amemory. The CPU is also referred to as a central processing device, aprocessing device, an arithmetic device, a microprocessor, amicrocomputer, a processor or a Digital Signal Processor (DSP).

FIG. 7A is a diagram showing the hardware configuration example in acase where the functions of the image acquisition unit 1, the areaextraction unit 2, the eyelid detection unit 3, the reliabilitycalculation unit 4, the determination unit 5 and the state determinationunit 6 are implemented by a processing circuit 100 as dedicatedhardware. What corresponds to the processing circuit 100 is, forexample, a single circuit, a composite circuit, a programmed processor,a parallel-programmed processor, an Application Specific IntegratedCircuit (ASIC), a Field Programmable Gate Array (FPGA) or anycombination thereof. The functions of the image acquisition unit 1, thearea extraction unit 2, the eyelid detection unit 3, the reliabilitycalculation unit 4, the determination unit 5 and the state determinationunit 6 may be implemented by a combination of two or more processingcircuits 100, or the functions of the respective units may beimplemented by one processing circuit 100.

FIG. 7B is a diagram showing the hardware configuration example in acase where the functions of the image acquisition unit 1, the areaextraction unit 2, the eyelid detection unit 3, the reliabilitycalculation unit 4, the determination unit 5 and the state determinationunit 6 are implemented by a CPU 102 which executes a program stored in amemory 101. In this case, the functions of the image acquisition unit 1,the area extraction unit 2, the eyelid detection unit 3, the reliabilitycalculation unit 4, the determination unit 5 and the state determinationunit 6 are implemented by software, firmware or a combination ofsoftware and firmware. The software and the firmware are each written asa program and stored in the memory 101. The CPU 102 reads out andexecutes a program stored in the memory 101 to thereby implement thefunctions of the image acquisition unit 1, the area extraction unit 2,the eyelid detection unit 3, the reliability calculation unit 4, thedetermination unit 5 and the state determination unit 6. Namely, theerroneous detection determination apparatus 10 has the memory 101 forstoring programs, etc. by which the processing of Steps ST1 to ST23shown in the flowcharts of FIG. 3, FIG. 9 and FIG. 10 to be describedlater, is performed as a result. Further, it can also be said that theseprograms are programs for causing a computer to execute procedures ormethods which the image acquisition unit 1, the area extraction unit 2,the eyelid detection unit 3, the reliability calculation unit 4, thedetermination unit 5 and the state determination unit 6 use. Here, whatcorresponds to the memory 101 is, for example, a non-volatile orvolatile semiconductor memory, such as a Random Access Memory (RAM), aRead Only Memory (ROM), a flash memory, an Erasable Programmable ROM(EPROM), an Electrically Erasable Programmable ROM (EEPROM) and thelike; or a disc-like memory medium, such as a magnetic disc, a flexibledisc, an optical disc, a compact disc, a mini disk, a Digital VersatileDisc (DVD) and the like.

It is noted that the functions of the image acquisition unit 1, the areaextraction unit 2, the eyelid detection unit 3, the reliabilitycalculation unit 4, the determination unit 5 and the state determinationunit 6 may be implemented partly by dedicated hardware and partly bysoftware or firmware. For example, it is allowed that, with respect tothe image acquisition unit 1, the area extraction unit 2 and the eyeliddetection unit 3, their functions are implemented by a processingcircuit as dedicated hardware, and with respect to the reliabilitycalculation unit 4, the determination unit 5 and the state determinationunit 6, their functions are implemented by causing a processing circuitto read out and execute a program stored in a memory.

In this manner, using hardware, software, firmware or any combinationthereof, the processing circuit can implement the functions of the imageacquisition unit 1, the area extraction unit 2, the eyelid detectionunit 3, the reliability calculation unit 4, the determination unit 5 andthe state determination unit 6.

Next, using the flowchart shown in FIG. 3, an example of processing bythe erroneous defection determination apparatus 10 configured asdescribed above will be described. The sequential processing shown inFIG. 8 is carried out periodically when, for example, the driver hasstarted driving the vehicle.

The image acquisition unit 1 acquires image data indicating an image ofthe driver (Step ST1). The image acquisition unit 1 outputs the acquiredimage data to the area extraction unit 2.

Then, using the image data acquired by the image acquisition unit 1, thearea extraction unit 2 extracts the eye area in the image indicated bythe image data (Step ST2).

The area extraction unit 2 outputs the partial image data correspondingto the eye area of the driver, to the eyelid detection unit 3 and thereliability calculation unit 4.

Then, using the image data outputted from the area extraction unit 2,the eyelid detection unit 3 detects an eyelid in the eye area extractedby the area extraction unit 2 (Step ST3).

The eyelid detection unit 3 outputs the positional information of thedetected eyelid to the reliability calculation unit 4 and the statedetermination unit 6.

Then, using the luminance information of the eye area extracted by thearea extraction unit 2 and the positional information of the eyeliddetected by the eyelid detection unit 3, the reliability calculationunit 4 calculates the eyelid reliability (Step ST4). The reliabilitycalculation unit 4 outputs the calculated eyelid reliability to thedetermination unit 5.

In this respect, in FIG. 9, an example of processing in Step ST4 isillustrated precisely.

First, the reliability calculation unit 4 sets the eyelid reliability toa maximum value as an initial value (Step ST11). When, for example, theeyelid reliability is expressed as a percent, the maximum value of theeyelid reliability is 100.

Then, the reliability calculation unit 4 calculates the eyelidevaluation value in the already described manner. The reliabilitycalculation unit 4 then determines whether the calculated eyelidevaluation value is equal to or greater than the fourth threshold value(Step ST12).

When the eyelid evaluation value is equal to or greater than the fourththreshold value (Step ST12; YES), the reliability calculation unit 4sets the eyelid reliability in a manner according to the eyelidevaluation value, as shown, for example, in FIG. 4 (Step ST13). When,for example, the eyelid reliability is expressed as a percent, theeyelid reliability to be set in Step ST13 is a value larger than zero.Accordingly, the eyelid reliability set in Step ST11 is updated.

In contrast, when the eyelid evaluation value is less than the fourththreshold value (Step ST12; NO), the reliability calculation unit 4 setsthe eyelid reliability to the minimum value (Step ST14). When, forexample, the eyelid reliability is expressed as a percent, the minimumvalue of the eyelid reliability is zero. Accordingly, the eyelidreliability set in Step ST11 is updated. Note that, as alreadydescribed, when the eyelid evaluation value is less than the fourththreshold value, the reliability calculation unit 4 may decrease,instead of uniformly setting the eyelid reliability to the minimumvalue, the eyelid reliability in a stepwise manner according to theeyelid evaluation value.

Returning back to FIG. 8, the determination unit 5 determines whetherthe eyelid reliability calculated in Step ST4 in the above manner isequal to or greater than the first threshold value (Step ST5).

When the eyelid reliability is equal to or greater than the firstthreshold value (Step ST5; YES), the determination unit 5 determinesthat the eyelid has been properly detected (Step ST6). The determinationunit 5 outputs the determination result to the state determination unit6.

The state determination unit 6 is in operation for calculating the eyeopen-closed state by using the positional information of the eyeliddetected by the eyelid detection unit 3. The state determination unit 6determines that the calculated open-closed state is valid (Step ST7).

In contrast, when the eyelid reliability is less than the firstthreshold value (Step ST5; NO), the determination unit 5 determines thatthe eyelid has not been properly detected (Step ST5). The determinationunit 5 outputs the determination result to the state determination unit6.

The state determination unit 6 is in operation for calculating the eyeopen-closed state by using the positional information of the eyeliddetected by the eyelid detection unit 3. The state determination unit. 6determines that the calculated open-closed state is invalid (Step ST9).

Note that the state determination unit 6 may be configured to calculatethe eye open-closed state when the eyelid has been determined to beproperly detected, and not to calculate the eye open-closed state whenthe eyelid has been determined to be not properly detected.

In this manner, the erroneous detection determination apparatus 10determines whether or not the eyelid has been erroneously detected. Evenwhen scenery reflected in an eyeglass lens surface around the eye asshown in FIG. 5B is detected as an eyelid, the erroneous detectiondetermination apparatus 10 can correctly determine that the eyelid hasbeen erroneously detected, so that accuracy is high in determining theerroneous detection.

Further, when, by the determination unit 5, the eyelid has beendetermined to be not properly detected, the eye open-closed statecalculated by the state determination unit 6 is determined to beinvalid. Accordingly, in the case where a state of the driver, such asfalling asleep, sleepiness, an eye blink and the like, is sensed usingthe eye open-closed state, sensing capability is improved.

Meanwhile, in order to prevent occurrence of the erroneous detection ofan eyelid due to reflection of scenery in the eyeglass lens surface, itis conceivable as a countermeasure, to cause the camera to performimaging in such a manner that no scenery is reflected in the eyeglasslens surface. However, in this case, it is necessary to intensify thecamera illumination at the time of imaging, so that the cost increases.According to the erroneous detection determination apparatus 10, even inthe case without such a countermeasure, it is possible to accuratelydetermine the erroneous detection of an eyelid, to thereby restrain thecost from increasing.

It is noted that the reliability calculation unit 4 may calculate theeyelid reliability according to previously calculated eyelidreliabilities.

An example of processing in this case will be described using theflowchart shown in FIG. 10. Note that, for the steps that performprocessing that is the same as or equivalent to the processing by thesteps already described in FIG. 9, the same numerals are given, so thatdescription thereof will be appropriately omitted or simplified.

The reliability calculation unit 4, every time it calculates an eyelidreliability in Step ST4, stores the calculated eyelid reliability in anunshown storage unit. As a result, a history of eyelid reliabilitiespreviously calculated by the reliability calculation unit 4 is stored inthe erroneous detection determination apparatus 10.

Subsequent to Step ST13 or Step ST14, the reliability calculation unit 4reads out a history of eyelid reliabilities from the unshown storageunit. At this time, the reliability calculation unit 4 reads out thehistory of eyelid reliabilities on a series of images from one frame tospecified frames before a target image for which the eyelid reliabilityhas been calculated in Step ST13 or Step ST14. Accordingly, one or aplurality of eyelid reliabilities is read out. When, for example, thespecified frames are five frames, the reliability calculation unit 4reads out five eyelid reliabilities corresponding to the images from oneframe to five frames before the target image for which the eyelidreliability has been calculated in Step ST13 or Step ST14.

Then, the reliability calculation unit 4 calculates a ratio, in theeyelid reliabilities thus read out, of eyelid reliabilities of less thana second threshold value (Step ST21). In this manner, the reliabilitycalculation unit 4 calculates a ratio of images each with the eyelidreliability of less than the second threshold value, in the imagesindicated by the image data previously acquired by the image acquisitionunit 1. Note that the second threshold value may be set to the samevalue as the first threshold value, or may be set to a value less thanthe first threshold value.

Then, the reliability calculation unit 4 determines whether thethus-calculated ratio is equal to or greater than a third thresholdvalue (Step ST22).

When the calculated ratio is equal to or greater than the thirdthreshold value (Step ST22; YES), the reliability calculation unit 4sets the eyelid reliability to a minimum value (Step ST23). Note that,at this time, since the reliability calculation unit 4 only has todecrease the eyelid reliability, the reliability calculation unit. 4 maydecrease the eyelid reliability to a value other than the minimum value.Accordingly, the eyelid reliability set in Step ST13 or Step ST14 isupdated and then outputted to the determination unit 5.

In contrast, when the calculated ratio is less than the third thresholdvalue (Step ST22; NO), the reliability calculation unit 4 makes nochange in the eyelid reliability set in Step ST13 or Step ST14. Namely,the eyelid reliability set in Step ST13 or Step ST14 is outputted to thedetermination unit 5.

As to the processing shown in FIG. 10, additional description will bemade using FIG. 11A, FIG. 11B, FIG. 12, FIG. 13 and FIG. 14.

FIG. 11A shows a case where, in an eyeglass lens surface, scenery isreflected in a band shape under the eye. In this case, out of the entireeyelids, only a portion surrounded by a broken line in the figure is tobe detected as an eyelid by the eyelid detection unit 3. When the eyelidis detected as described just above, unlike the actual state of the eye,the eye-opening degree is calculated to be low, so that the eye isregarded as if it were nearly closed. Namely, the eyelid detection unit3 has erroneously detected the eyelid. However, since the normal vectorsVa and the luminance vectors Vb are well-matched to each other, theeyelid reliability calculated by the reliability calculation unit 4 ishigh, so that the determination unit 5 cannot determine that suchdetection is erroneous detection.

Further, FIG. 11B shows a case where, in a state in which an eye isclosed, scenery having a contour similar to an eye is reflected in theeyeglass lens surface. In this case, an edge portion of the scenerysurrounded by a broken line in the figure is to be detected as eyelidsby the eyelid detection unit 3. In this case also, since the normalvectors Va and the luminance vectors Vb are well-matched to each other,the eyelid reliability calculated by the reliability calculation unit 4is high, so that the determination unit 5 cannot determine that suchdetection is erroneous detection.

FIG. 12 is a diagram showing an example of temporal transition ofscenery reflection and eyelid reliability. Let's assume that the sceneryreflection has varied, starting from State T1 to reach State T6 as shownside by side along the horizontal axis. In FIG. 12, eyelid reliabilitiesare shown that have been calculated for the images reflecting therespective States T1 to T6, by the reliability calculation unit 4 inaccordance with the processing shown in FIG. 9. At each of States T1 andT5, since there is no scenery reflection, the eyelid reliability iscalculated to foe high. At each of States T2, T3 and T6, since sceneryis reflected entirely in the eyeglass lens surface, the eyelidreliability is calculated to be low. At State T4, as has been describedusing FIG. 11A, even though the eyelid detection unit 3 has erroneouslydetected the eyelid, the eyelid reliability is calculated to be high.

FIG. 13 is a diagram corresponding to the case where scenery reflectionsshown in FIG. 12 have occurred, and showing an example of temporaltransition of a ratio of images each with the eyelid reliability of lessthan the second threshold value, in previous images. For example, theratio at State T3 is provided as a ratio of images each with the eyelidreliability calculated to be less than the second threshold value, inthe images of specified frames at and prior to State T2. In accordancewith the processing shown in FIG. 10, when the transition of the ratiois as shown in FIG. 13, the eyelid reliability is set to the minimumvalue at each of States T3 and T4. Namely, as shown in FIG. 14, theeyelid reliability is low also at State T3. Accordingly, with respect toState T3, the determination unit 5 can determine that the eyelid hasbeen erroneously detected.

In this manner, at the time the reliability calculation unit 4calculates the eyelid reliability, not only the image currently subjectto processing but also the images previously subject to processing aretaken into consideration, so that accuracy is more improved indetermining the erroneous detection.

It is noted that, in the foregoing, a case is shown where the erroneousdetection determination apparatus 10 is installed in a vehicle. However,the image acquisition unit 1, the area extraction unit 2, the eyeliddetection unit 3, the reliability calculation unit 4, the determinationunit 5 and the state determination unit 6, which are included in theerroneous detection determination apparatus 10, may be configured in anunshown server outside the vehicle. In this case, the unshown serveracquires image data indicating an image of the driver through wirelesscommunication from the vehicle, to thereby determine the erroneousdetection of an eyelid. Further, on the basis of the calculated eyeopen-closed state, the unshown server transmits, where necessary, asignal for informing the driver of a warning, to the vehicle. In thismanner, a server may function as the erroneous detection determinationapparatus 10.

Further, in the foregoing, although the erroneous detectiondetermination apparatus 10 is installed in a vehicle, the erroneousdetection determination apparatus 10 may instead be a smartphone or thelike of the driver or the like, brought into a vehicle. In this case,the smartphone acquires image data indicating an image of the driver, byusing an unshown camera for imaging the driver installed in the vehicle,or a camera built in the smartphone.

Further, for example, it is allowed that the eyelid detection unit 3 andthe reliability calculation unit 4 are configured in an unshown serveroutside the vehicle and thereby the unshown server performs detection ofthe eyelid and calculation of the eyelid reliability. In this case, theunshown server acquires image data indicating the eye area throughwireless communication from the vehicle, to thereby determine theerroneous detection of an eyelid. Then, the unshown server transmits thepositional information of the detected eyelid and the calculated eyelidreliability to the vehicle. In this manner, the units that are includedin the erroneous detection determination apparatus 10 may be distributedat different places such as a vehicle and a server outside the vehicle.

Further, in the foregoing, the description has been made citing thedriver of a vehicle as an example; however, the erroneous detectiondetermination apparatus 10 may be used for a driver/operator of a movingobject other than a vehicle, as a target.

Further, the erroneous detection determination apparatus 10 may be thatwhich outputs, without having the state determination unit 6, aprocessing result to an external apparatus. Namely, the erroneousdetection determination apparatus 10 outputs the positional informationof the eyelid detected by the eyelid detection unit 3 and thedetermination result by the determination unit 5, to the statedetermination unit 6 provided outside the erroneous detectiondetermination apparatus 10. Upon receiving the output from the erroneousdetection determination apparatus 10, the state determination unit 6provided outside the erroneous detection determination apparatus 10calculates the eye open-closed state, to thereby sense a state of thedriver, such as falling asleep, sleepiness, an eye blink and the like.

As described above, according to Embodiment 1, since the reliabilitycalculation unit 4 calculates the eyelid reliability by using theluminance information of the eye area and the positional information ofthe eyelid, and the determination unit 5 determines using the eyelidreliability whether or not the eyelid has been erroneously detected, itis possible to improve accuracy in determining the erroneous detection.

In another aspect, the erroneous detection determination apparatus 10 isprovided with the state determination unit 6 for calculating the eyeopen-closed state by using the positional information of the eyeliddetected by the eyelid detection unit 3, and when the determination unit5 determines that the eyelid has not been properly detected, the statedetermination unit 6 determines that the calculated open-closed state isinvalid. Accordingly, in the case where a state of the driver, such asfailing asleep, sleepiness, an eye blink and the like, is sensed usingthe eye open-closed state, sensing capability is improved.

In another aspect, when, in the images indicated by the respectivepieces of image data having been acquired by the image acquisition unit1, the ratio of images each with the eyelid reliability of less than thesecond threshold value, is not less than the third threshold value, thereliability calculation unit 4 decreases the eyelid reliability that iscalculated by using the luminance information and the positionalinformation of the eyelid, and the determination unit 5 performs thedetermination by using the eyelid reliability decreased by thereliability calculation unit 4. Accordingly, accuracy is more improvedin determining the erroneous detection.

In another aspect, the reliability calculation unit 4 calculates theeyelid reliability by using the inner products of the respectiveluminance vectors obtained from the luminance information and therespective normal vectors of an eyelid line obtained from the positionalinformation. Accordingly, it is possible to calculate the eyelidreliability, adequately.

It should be noted that modification of any component in the embodimentand omission of any component in the embodiment may be made in thepresent invention without departing from the scope of the invention.

INDUSTRIAL APPLICABILITY

As describe above, the erroneous detection determination apparatusaccording to the invention can improve accuracy in determining whetheror not the eyelid has been erroneously detected, and is thus suited foruse in a manner incorporated in a system for performing processing upondetection of an eyelid, for example, a driver monitoring system.

REFERENCE SIGNS LIST

1: image acquisition unit, 2: area extraction unit, 3: eyelid detectionunit, 4: reliability calculation unit, 5: determination unit, 6: statedetermination unit, 10: erroneous detection determination apparatus,100: processing circuit, 101: memory, 102: CPU.

1-5. (canceled)
 6. An erroneous detection determination apparatus,comprising: processing circuitry to acquire image data indicating animage of a driver; to extract an eye area in the image by using theimage data acquired; to detect an eyelid in the eye area extracted; tocalculate an eyelid reliability by using luminance information of theeye area extracted and positional information of the eyelid detected;and to determine that the eyelid has not been properly detected, whenthe eyelid reliability calculated is less than a first threshold value,wherein the processing circuitry calculates the eyelid reliability byusing inner products of respective luminance vectors obtained from theluminance information and respective normal vectors of an eyelid lineobtained from the positional information.
 7. The erroneous detectiondetermination apparatus of claim 6, wherein the processing circuitrycalculates an eye open-closed state by using the positional informationof the eyelid detected, and when it is determined that the eyelid hasnot been properly detected, determines that the calculated open-closedstate is invalid.
 8. The erroneous detection determination apparatus ofclaim 6, wherein when, in images indicated by respective pieces of imagedata having been acquired by the processing circuitry, a ratio of imageseach with the eyelid reliability of less than a second threshold value,is not less than a third threshold value, the processing circuitrydecreases the eyelid reliability that is calculated by using theluminance information and the positional information of the eyelid, andperforms the determination as to whether the eyelid has not beenproperly detected, by using the eyelid reliability decreased.
 9. Theerroneous detection determination apparatus of claim 7, wherein when, inimages indicated by respective pieces of image data having been acquiredby the processing circuitry, a ratio of images each with the eyelidreliability of less than a second threshold value, is not less than athird threshold value, the processing circuitry decreases the eyelidreliability that is calculated by using the luminance information andthe positional information of the eyelid, and performs the determinationas to whether the eyelid has not been properly detected, by using theeyelid reliability decreased.
 10. An erroneous detection determinationapparatus, comprising: processing circuitry to acquire image dataindicating an image of a driver; to extract an eye area in the image byusing the image data acquired; to detect an eyelid in the eye areaextracted; to calculate an eyelid reliability by using luminanceinformation of the eye area extracted and positional information of theeyelid detected; to determine that the eyelid has not been properlydetected, when the eyelid reliability calculated is less than a firstthreshold value; and to calculate an eye open-closed state by using thepositional information of the eyelid detected, wherein when it isdetermined that the eyelid has not been properly detected, theprocessing circuitry determines that the calculated open-closed state isinvalid, and when, in images indicated by respective pieces of imagedata having been acquired by the processing circuitry a ratio of imageseach with the eyelid reliability of less than a second threshold value,is not less than a third threshold value, the processing circuitrydecreases the eyelid reliability that is calculated by using theluminance information and the positional information of the eyelid, andperforms the determination as to whether the eyelid has not beenproperly detected, by using the eyelid reliability decreased.
 11. Theerroneous detection determination apparatus of claim 10, wherein theprocessing circuitry calculates the eyelid reliability by using innerproducts of respective luminance vectors obtained from the luminanceinformation and respective normal vectors of an eyelid line obtainedfrom the positional information.
 12. An erroneous detectiondetermination method, comprising: acquiring image data indicating animage of a driver; extracting an eye area in the image by using theimage data acquired; detecting an eyelid in the eye area extracted;calculating an eyelid reliability by using luminance information of theeye area extracted and positional information of the eyelid detected;and determining that the eyelid has not been properly detected, when theeyelid reliability calculated is less than a first threshold value,wherein the eyelid reliability is calculated using inner products ofrespective luminance vectors obtained from the luminance information andrespective normal vectors of an eyelid line obtained from the positionalinformation.